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ABSTRACT 

In this article, we introduce readers to a special issue of Studies in 
Conflict & Terrorism on measurement issues in the study of terrorism. In 
recent years scholarly interest in terrorism has increased and 
systematic methods are now more commonly used. Terrorism works 
that analyze data highlight substantive findings as opposed to 
measurement issues. A study's substantive findings are only 
meaningful though if it correctly addresses the measurement issues 
that invariably arise during the research process. In addition to 
summarizing the eight articles in this special issue, we outline the 
strengths and weaknesses of various measurement strategies and 
assess their contributions to our understanding of terrorism. The major 
goal is to place the special issue’s contributions in context and 
highlight under-explored issues that future research could address. 


This special issue of Studies in Conflict & Terrorism focuses on measurement issues in the 
study of terrorism, an important but understudied topic. Recently, scholarly interest in ter- 
rorism has increased and systematic methods are now more commonly used. Importantly 
though, terrorism works continue to lag behind related fields of study in the analysis of data 
and the adoption of sophisticated research methods.’ The few terrorism works that do ana- 
lyze data highlight substantive findings as opposed to measurement issues. A study’s sub- 
stantive findings are only meaningful though if it correctly addresses the measurement 
issues that invariably arise during the research process. In other words, while the quality of 
terrorism research has greatly improved in recent years, measurement issues have been 
under explored and researchers could do more in this area to further improve the field’s 
rigor. 

This special issue hopes to begin to fill this gap. The eight articles in this issue raise, dis- 
cuss, and provide solutions for methodological issues that could make important contribu- 
tions to the social scientific study of terrorism. We begin by outlining some general points 
about the relationship between the strengths and weaknesses of various measurement 
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strategies and our understanding of terrorism. The goal is to place the special issue’s contri- 
butions in context and highlight under-explored issues that future research could address. 

Almost all discussions on measuring terrorism begin by noting the difficult conceptual 
and methodological issues that must be confronted. The definitional debate, of course, looms 
large.” There is no universally accepted terrorism definition by most countries, the United 
Nations, academics, or policymakers. Indeed, operational definitions even vary within differ- 
ent branches of the U.S. government. Scholars have empirically demonstrated that these def- 
initions diverge on whether they include or exclude each of 22 distinct elements.’ Examples 
of disagreement include whether the act must target civilians, be random, or intended to 
cause fear in the population. Nonetheless, currently almost all terrorism definitions require 
the act to be violent, ideologically motivated and committed by non-state actors.* 

Historically, terrorism researchers rarely used statistical methods in their studies.” Silke’s 
extensive review of research published in the major terrorism journals between 1995 and 
2000,° and Lum, Kennedy, and Sherley’s Campbell subsequent systematic review’ of over 
14,000 terrorism articles published between 1971 and 2003 both found that only three per- 
cent of studies employed statistical analyses. This was much lower compared to the related 
fields of forensic psychology and criminology. It could be that terrorism scholars have less 
access than criminologists and forensic psychologists to data sources that could be used for 
quantitative analyses. 

Criminologists often use official data collected and maintained by the police and other 
government agencies. One prominent example of official crime data are the United States 
government’s Federal Bureau of Investigation (FBI)’s Uniform Crime Reports (UCR). The 
UCR is a voluntary program that publishes a large number of crime statistics based on infor- 
mation submitted by local police agencies that originate from jurisdictions covering most of 
the U.S. population. Another data source widely used by criminologists is information 
obtained from crime victims. For instance, the Census Bureau collects and the Department 
of Justice administers the National Crime Victimization Survey (NCVS), a national house- 
hold survey that asks respondents if they have been victimized by many of the same crimes 
tracked by the UCR. Both the UCR and the NCVS data are widely used by scholars to study 
the etiology of ordinary street crime. 

LaFree and Dugan® and Dugan and Distler? point out, however, that it is difficult to rely 
on victim or police data to study terrorism. In many countries the rate of terrorism is low, 
and it would be difficult to create a nationally representative sample of terrorism victims. A 
strategy that relied on victimization data by definition would exclude the most severe terror- 
ist acts (where victims perished). Another reason why terrorism researchers rarely focus on 
victims is because it is assumed that there is no variation to investigate. In other words, it is 
thought that terrorism victims are randomly selected’? Parkin, Freilich, and Chermak’s pre- 
liminary examination, however, found that over 95 percent of the victims of far-right ideo- 
logically motivated homicide attacks in the United States were not randomly selected. 
Instead they were purposefully selected (such as the victims of assassinations) or selected 
because of what they represented (such as victims targeted because of their perceived racial, 
religious or other minority group membership).'’ Nonetheless, and perhaps not surprisingly 
then, few terrorism studies have used victimization data.'* 

There are also difficulties in relying on the police for terrorism data. Some terrorism 
attacks may not be reported to the police or not recorded in their data systems because they 
fail to meet the inclusion criteria and are subsumed under another crime type. Relatedly, 
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terrorist acts are not always prosecuted as terrorism but instead as ordinary crime.'* The 
inherently political nature of terrorism may also influence government tracking of its occur- 
rence. Some researchers are thus reticent to use police data. In addition, the government 
may be equally hesitant to share their terrorism data with academics for both political and 
national security reasons. 

Terrorism researchers are therefore less likely than criminologists studying ordinary 
crime to use police or other government data. One noted exception is the important data col- 
lection effort conducted by Brent Smith and his colleagues for the American Terrorism 
Study (ATS).'* The ATS has been conducted in cooperation with the FBI’s Terrorist 
Research and Analytical Center, and includes persons indicted federally as a result of investi- 
gations under the FBI’s Counterterrorism Program. Smith and colleagues were provided lists 
of individuals indicted for terrorist offenses, and they then traveled to federal courthouses to 
collect data from trial transcripts and docket information. Smith and colleagues have used 
these data to answer important questions, including the effects of political motive on sen- 
tencing outcomes,’” profiles of American terrorists,'° and prosecutorial strategies in terror- 
ism cases.'” 

Other scholars have also managed to overcome obstacles and obtain terrorism data from 
official sources including the police or military.'* Two studies in this special issue follow in 
this tradition by obtaining terrorism data from the police to engage important methodologi- 
cal issues. Behlendorf, Belur, and Kumar use data from the Andhra Pradesh Police, a state in 
India to make a comparison to three media-based datasets that also tracked terrorism inci- 
dents in this state. Similarly, De Bie and De Poot obtained unique Dutch data that included 
police and court files containing wiretaps of both telephone and Internet communications, 
recordings of in-house communication, transcripts of suspect interrogations, witness state- 
ments, observation reports, forensic reports, reports of house searches, and expert-witness 
reports. Unlike Behlendorf and colleagues who employed quantitative methods, De Bie and 
De Poot used their exceedingly rich official data to illustrate how grounded theory-based 
qualitative methods can be used for analysis. Their goal was to develop inductively new theo- 
ries of terrorist etiology. 

Another major strategy criminologists employ to measure regular crime is obtaining 
information from the offenders themselves through interviews. This approach is referred to 
by criminologists as “self-reports.” Criminologists have analyzed self-report data both quan- 
titatively and qualitatively. In fact, some classic criminological research hails from the genre 
of qualitative self-report studies.'? There are also large scale criminology and delinquency 
data collection efforts that interview youths to collect information to code attributes about 
both the offenders and the crimes they commit.”° Sometimes these efforts focus on the same 
type of crimes tracked by the UCR and the NCVS. Many criminology studies currently 
appearing in leading journals use quantitative self-report data.” The quantitative self-reports 
often only ask about specific offending patterns for a distinct period (such as six months or a 
year) to minimize telescoping (incorrectly including or excluding into the reference period 
asked about). 

Because the use of official and victimization data by terrorism scholars is limited due to 
the obstacles outlined above, many terrorism scholars have embraced the self-report strat- 
egy. Silke’s review demonstrated that the interview method is one of the more popular data 
sources and methods employed by terrorism researchers.”” Despite the common refrain that 
it is difficult to talk to terrorists due to safety and access issues, scholars have actually 
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interviewed (former and current) terrorists from almost every continent and extremist ideo- 
logical belief system.”° Silke concluded that the interview method is flexible and it provides 
researchers a measure of control.”* 

However, Silke also noted that the interview method potentially suffers from weaknesses. 
Limitations include gaining access to terrorists, its expense, interviewer bias, generalizability, 
opportunity sampling, and interviewer effects. Importantly, most terrorism interviews are 
retrospective that raises the possibility that they suffer from retrospective construction; the 
human tendency to construct specific moments as significant in retrospect. 

Criminologists have responded to these measurement challenges in a variety of ways. For 
example, some have oversampled youths from high crime areas to address low base rates, 
and sampling issues. Others have used longitudinal designs, bounded responses by conduct- 
ing repeat interviews and minimized retrospective construction by focusing on short refer- 
ence periods.” 

Most terrorism studies that employ the interview method, however, fail to note systemati- 
cally these methodological challenges and rarely explain what steps, if any, were taken to 
overcome them. In addition, most interview studies provide little detail on such potentially 
important issues as the interview setting, or the specific nature of the interview protocols 
used.*° Harris, Simi, and Ligon’s contribution to this special issue engages these potential 
limits of surveys by investigating the methodological transparency of the interview method 
in terrorism research. They note that an increasing number of terrorism journal articles use 
the interview method. Harris and colleagues content analyzed 48 articles involving such 
interviews, and conclude that methodological transparency in these studies is often lacking. 
The authors then provide recommendations to enhance methodological transparency. 

One of the major reasons for the increased use of statistical analysis in terrorism studies 
has been the creation and increasing availability of terrorism event, offender, and organiza- 
tional-level databases. For example, the Global Terrorism Database (GTD),”” International 
Terrorism: Attributes of Terrorist Events (ITERATE) database,* the Minorities at Risk 
Organizational Behavior (MAROB) dataset,’ and the United States Extremist Crime Data- 
base (ECDB),*’ have all been used to conduct quantitative terrorism studies to investigate 
important theoretical and policy relevant issues. All these databases were built by using open 
sources”! 

The creation and growth of the Internet has made it easier for scholars to rely upon open 
sources to track systematically various terrorism attributes. Chermak and colleagues explain 
that open-source data refer to information that is open to public.*” Often this information is 
available electronically and online. Open-source data have similarities with secondary analy- 
sis because much of it was not initially collected for research purposes. Examples of open- 
source information include media reports, books, some types of court records, watch-group 
materials, information produced by the extremist (or terrorist) group/movement, as well as 
other publicly available information. 

Quantitative terrorism studies that rely on open sources also potentially suffer from meth- 
odological problems. For example, the failure of most studies to use non-terrorism or nonvi- 
olent control groups has hindered the advance of terrorism research.” It is not always clear 
though who should comprise the non-terrorist comparison group. The few prior studies that 
use comparison groups have relied upon a variety of strategies. Some, like Smith and Damp- 
housse, and Gruenewald and Pridemore have used violent non-extremist regular offenders.** 
Others have employed nonviolent extremist offenders. Kerodal, Freilich, and Chermak’s 
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intriguing contribution to this special issue, for instance, examines both violent far-right 
extremists (akin to most terrorists) as well as far-rightists who only committed nonviolent 
financial crimes. Alternative strategies include comparing violent (terrorist) extremists to 
non-criminal extremists” or to the general population or the terrorists’ reference group in 
the general population.*° Other strategies are also possible such as disaggregating different 
categories of terrorists and comparing them to each other.*” 

Another measurement issue that open-source terrorism studies commonly face is 
source type reliability. Sometimes the materials found on a specific case include different 
types of sources (e.g., indictments versus news articles) that contain conflicting values 
for the same attribute. Such discrepancies call for ranking the different source types in 
terms of reliability to resolve these conflicts.°* Ackerman and Pinson’s impressive study 
in this special issue addresses the reliability and validity of the open sources used to cre- 
ate open-source databases. They developed a sample Source Evaluation Schema to opera- 
tionalize measures of open-source event validity at the case, source and variable levels. 
Ackerman and Pinson conclude that explicitly including credibility and validity levels for 
open sources will provide greater flexibility in tailoring the inclusion of cases for 
researchers’ specific analytical requirements. Similar to Harris and colleagues’ conclu- 
sions regarding methodological transparency and the interview method, Ackerman and 
Pinson argue that facilitating more transparent analyses of the open-source databases by 
including such measures can result in more defensible conclusions. This is especially 
important in the highly charged political and security context of terrorism. 

Relatedly, coder reliability issues always arise when evaluating open-source databases. 
Many of these databases are large-scale efforts that often use computer algorithms and mul- 
tiple persons to identify and code incidents. Thus, tests of the interrater reliability of items 
included in these terrorism databases is an increasingly important issue that should be 
addressed.*” It should be noted though, that unlike studies that rely upon static data, most 
terrorism databases are dynamic and are updating variables as new information emerges or 
to reflect changes that have occurred. While many open-source terrorism studies have not 
always discussed the issue of coder reliability, a few preliminary efforts have moved in this 
direction. 

The ECDB’s initial examination of coder reliability for certain situational attributes of 
homicide events found coder agreement of 90 percent and higher. The ECDB trains its 
coders, constantly compares coding across coders to identify any abnormalities, and uses a 
coder listserv to discuss complicated issues that arise. When ECDB coders diverged it was 
most often because one filled in an attribute while the other did not and left it as missing, as 
opposed to the two coders inputting conflicting values. Freilich and colleagues concluded 
that one coder had located a source that contained the necessary information, while the 
other coder did not. They therefore suggested that multiple coders should search and code 
each incident to insure that all relevant documents were located.*° 

The Profiles of Individual Radicalization in the United States (PIRUS) database includes 
over 120 variables on nearly 1,500 individuals who have radicalized in the United States.** 
PIRUS data are based on publicly available court documents, newspaper accounts, and pub- 
lished sources on individuals drawn generally from three ideological groups: far left, far 
right, and radical Islamic. The data include all individuals who have radicalized within the 
United States to the point of committing ideologically motivated illegal violent or nonviolent 
acts, joining a designated terrorist organization, or associating with an organization whose 
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leader(s) has/have been indicted of an ideologically motivated violent offense. To check the 
reliability of the coding the researchers took a 10 percent random sample of cases, coded 
each case twice using separate individuals and used Krippendorff’s alpha procedure to assess 
the reliability of the coding procedures.*” But while we are beginning to see these kinds of 
coder reliability tests in the open-source literature on terrorism, it is still relatively 
uncommon. 

Another open-source database issue relates to the missing values of variables. Some open- 
source databases have had great success in minimizing missing values. Parkin describes an 
innovative strategy, for example, that used highly targeted follow-up searches of public 
records to limit missing data on key variables to than 1 percent of the population. ** He used 
sources like state, local, and federal inmate locators; online local court dockets; the social 
security death index; and online national record aggregators such as Ancestry.com, 
Archives.com, BeenVerified.com and other news aggregates to collect information on births, 
marriages, deaths, military service, address histories, relatives, associates, criminal histories, 
filed bankruptcies, and financial liens and judgments. All of this information is greatly useful 
in reducing missing information. 

One fundamental point is whether an attribute (e.g., whether an offender had a criminal 
history) should be coded as “no” versus as “missing” if the open sources provide no informa- 
tion on it. Parkin and Freilich argue that in many of these cases if no information is found 
one way or the other on the attribute, the default coding should be “0” (for example, “no” 
criminal history) as opposed to missing. “* They maintain that this is necessary when work- 
ing with open-source materials where information may be left out because it is not consid- 
ered relevant to the story. Parkin and Freilich state that most sources, especially media 
outlets, would have no reason to report a negative response (e.g., an offender was never 
arrested), but would be much more likely to report if the offender did have such an attribute. 
They also point out that if scholars disaggregate the open-source data they are using (such as 
categorizing all ECDB or GTD terrorist incidents as suicide attacks, or non-suicide attacks) 
then both categories would be identified, searched, and coded using the same methods. This 
includes the default strategy of coding cases with no information on an attribute as no, 
instead of missing. Thus, there is no reason to believe that the likelihood of missing data 
would systematically vary between these two groups (e.g., suicide versus non-suicide attacks) 
if the default strategy they suggest is used. 

In a recent analysis of the PIRUS data, Jasko, LaFree, and Kruglanski*” used multiple 
imputation methods to replace missing data.*° In this approach instead of filling in missing 
observations with one specific value, the researchers created 25 datasets with a range of val- 
ues. Each imputed data set was then analyzed separately and the results of these separate 
analyses were pooled and averaged to obtain final coefficients. While promising this type of 
multiple imputation strategy is still rare in the analysis of open-source data on terrorism. 

There have also been few open-source studies that have looked at selectivity bias or cre- 
ated error profiles (i.e., listing non-sampling errors that could occur in each step of the data- 
base’s creation).*” The article by Behlendorf, Belur, and Kumar responds to this important 
critique by comparing terrorist attacks derived from three open-source media-based datasets 
(including the GTD and two other sources) and official police records from Andhra Pradesh 
state in India. Behlendorf and colleagues find that the media-based datasets capture the geo- 
graphic prevalence of terrorism yet severely underestimate the frequency of attacks, biasing 
toward lethal bombings. The authors conclude that existing terrorism databases include a 
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select version of violence in these countries, discounting especially the prevalence and regu- 
larity of non-lethal violent activity. 

Although, as noted, open-source databases include a variety of sources, the most common 
type relied on is media reports. Several years ago Chermak and Gruenewald investigated the 
media’s portrayal of terrorism inside the United States.** Parkin and Green’s intriguing con- 
tribution to the current issue follows in this tradition. They highlight the most efficient 
methodology for sampling from a population of New York Times articles related to terror- 
ism, which were generated through keyword searching. Efficiency was based on which sam- 
ple statistic was closest to the population parameters of interest. The smallest sample size, 
where 68 percent of the sample statistics were within one standard deviation of the popula- 
tion mean and 95 percent of the sample statistics were within two standard deviations of the 
population mean, were identified as the most efficient. They also determine whether the fre- 
quency of news articles is correlated with the temporal distribution of terrorist attacks found 
in the GTD to sample more efficiently from the population. Parkin and Green confirm prior 
research in that sampling efficiency is related to the weekly news cycle. But contrary to prior 
research, they also find that the sample must include between 20 to 29 constructed weeks to 
achieve representativeness of an entire year of coverage for a population generated through 
keyword searches. In addition, the study also finds that there is a limited relationship 
between the frequency of terrorist attacks and the amount of terrorism coverage in the news. 

It is also important to note that open-source event databases have overwhelmingly treated 
terrorist attacks and those who perpetrate them as if they were binary variables: either pres- 
ent or absent. But of course there is often definitional overlap between terrorism and other 
forms of crime and political violence, including insurgency, conflict between rival groups, 
hate crime, and organized crime. Moreover, in some cases the available information is insuf- 
ficient or conflicting. In response to these concerns, the GTD in recent years has added a fil- 
ter called “doubt terrorism proper,” which signifies that while the analysts’ concerns are not 
sufficient to exclude the case, they have enough uncertainty about the case to single it out.” 

Similarly, empirical studies commonly assume that terrorists all subscribe to the same 
ideological beliefs with the same intensity as others in their ideological group. Kerodal, Frei- 
lich, and Chermak’s contribution to this special issue explores this possibility by asking 
whether certain types of far-right (FR) ideological beliefs are associated with different types 
of criminal behavior. They also investigate if the various indicators of FR ideology can be 
used to create a scalar measure of commitment to FR ideology. They used ECDB data to 
examine a sample of far-rightists who committed a financial crime or homicide in the United 
States. They found that conspiratorial, antigovernment and antitax beliefs were positively 
associated with risk of financial crimes, while xenophobic, survivalist, and antigun control 
beliefs were positively associated with risk of violent crimes. A factor analysis created a com- 
mitment to FR ideology scale and identified four sub-types of FR extremists: Conspiracy 
Theorist, Survivalist, Movement Participant, and Proud Far-Rightists. The authors found 
that these typologies were useful in predicting criminal behavior patterns of far-rightists. 

Boyd’s contribution to this issue extends the literature by using multilevel modeling to 
assess both country characteristics, and group traits on the rate of domestic attacks and 
the rate of attacks against foreign targets. Prior research has only analyzed how country 
characteristics affect the rate of terrorist violence and there is a growing literature on 
how group traits influence terrorist violence. Boyd uses data from the Big Allied and 
Dangerous (BAAD) dataset and the GTD, to model a cross-national sample of 224 
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terrorist groups in relation to their countries of origin to assess rates of domestic attacks. 
In this cross-sectional study many of these terrorist groups target multiple foreign coun- 
tries. Multiple membership random effects modeling is used to assess the impact of mul- 
tiple countries targeted by a group. The results of the study show that multilevel 
modeling provides an improved statistical fit over more conventional methods and the 
modeling strategy used here provides an improved measurement for the data analyzing 
attacks targeting foreign countries. 

This special issue’s final contribution represents the terrorism field’s growing methodo- 
logical innovativeness in addressing important questions. Cohen examines Palestinian sui- 
cide attackers. Previous empirical studies focused on suicide terrorism have used media 
accounts, official data, interviews of failed suicide attackers, interviews of the handlers of sui- 
cide perpetrators, or examined the videos released by organizations after suicide attacks 
where attackers explain why they launched the strike.°’ Cohen’s study introduces a novel 
data source and methodology to further engage this issue. He obtained Palestinian suicide 
bombers’ personal farewell letters (which also serve as last wills) to their families and loved 
ones which allowed him to create a comprehensive corpus of suicide letters. To minimize 
bias, Cohen used a programmatic sequence of quantitative psycholinguistic procedures, in 
which every step informed the next one. His analysis of the cognitive content of the suicide 
bombers’ letters allows him to examine general themes and also test specific hypotheses 
regarding the motives and concerns of the attackers. 

Cohen’s research builds on prior studies that have applied linguistic content analyses of 
materials written by terrorists to identify possible associations with real world terrorist 
attacks.’ His results are consistent with theories of political violence that emphasize pro- 
social meaning making as a motivation for suicide terrorism, rather than antisocial senti- 
ments such as hatred and revenge. Cohen’s methods provide an important new tool for 
understanding individual motivations for suicide terrorism and for identifying those ele- 
ments in the organization’s ideology that were endorsed by the individual bombers. These 
fault lines could also have important implications for designing more effective countermes- 
saging campaigns. 

Terrorism researchers have noted for years that studies of terrorism have been slow to 
adopt the most recent methodological innovations. In this special issue we have attempted 
to address this criticism by providing a set of articles that extend the traditional methodolog- 
ical boundaries of terrorism research. Taken together, the eight articles in this special issue 
break important new methodological ground in the study of terrorism and we hope will 
encourage researchers to push boundaries even further in the years ahead. 
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